Anomaly Detection in Time Series

Tensorflow version: 2.1.0

S&P 500 Index Data

Data Source: S&P500 Daily Prices 1986 - 2018

date close
0 1986-01-02 209.59
1 1986-01-03 210.88
2 1986-01-06 210.65
3 1986-01-07 213.80
4 1986-01-08 207.97
(8192, 2)

Data Preprocessing

(6553, 2) (1639, 2)

Training and Test Splits

(6523, 30, 1)

LSTM Autoencoder

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm (LSTM)                  (None, 128)               66560     
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
repeat_vector (RepeatVector) (None, 30, 128)           0         
_________________________________________________________________
lstm_1 (LSTM)                (None, 30, 128)           131584    
_________________________________________________________________
dropout_1 (Dropout)          (None, 30, 128)           0         
_________________________________________________________________
time_distributed (TimeDistri (None, 30, 1)             129       
=================================================================
Total params: 198,273
Trainable params: 198,273
Non-trainable params: 0
_________________________________________________________________

Training the Autoencoder

Train on 5870 samples, validate on 653 samples
Epoch 1/100
5870/5870 [==============================] - 12s 2ms/sample - loss: 0.1625 - val_loss: 0.1610
Epoch 2/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.1114 - val_loss: 0.0986
Epoch 3/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0903 - val_loss: 0.0443
Epoch 4/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0802 - val_loss: 0.0442
Epoch 5/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0717 - val_loss: 0.0639
Epoch 6/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0776 - val_loss: 0.0327
Epoch 7/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0750 - val_loss: 0.0313
Epoch 8/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0744 - val_loss: 0.0578
Epoch 9/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0758 - val_loss: 0.0522
Epoch 10/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0765 - val_loss: 0.0302
Epoch 11/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0734 - val_loss: 0.0610
Epoch 12/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0749 - val_loss: 0.0629
Epoch 13/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0751 - val_loss: 0.0274
Epoch 14/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0746 - val_loss: 0.0407
Epoch 15/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0741 - val_loss: 0.0454
Epoch 16/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0762 - val_loss: 0.0277

Plotting Metrics and Evaluating the Model

1609/1609 [==============================] - 1s 374us/sample - loss: 0.2685
0.26845344528787396

Detecting Anomalies in the S&P 500 Index Data

date close loss threshold anomaly
8187 2018-06-25 4.493228 0.638412 0.65 False
8188 2018-06-26 4.507583 0.691408 0.65 True
8189 2018-06-27 4.451431 0.696458 0.65 True
8190 2018-06-28 4.491406 0.727353 0.65 True
8191 2018-06-29 4.496343 0.709381 0.65 True
date close loss threshold anomaly
7474 2015-08-25 2.457439 0.655041 0.65 True
7475 2015-08-26 2.632149 0.711078 0.65 True
8090 2018-02-05 4.329949 0.657327 0.65 True
8091 2018-02-06 4.440671 0.846347 0.65 True
8092 2018-02-07 4.408365 0.822247 0.65 True
date close loss threshold anomaly
8145 2018-04-25 4.307086 0.653256 0.65 True
8188 2018-06-26 4.507583 0.691408 0.65 True
8189 2018-06-27 4.451431 0.696458 0.65 True
8190 2018-06-28 4.491406 0.727353 0.65 True
8191 2018-06-29 4.496343 0.709381 0.65 True